Trading on the Edge/Trading On The Edge - CD-ROM Toolkit (Wayzata Technology)(2031)(1994).bin/pc/mac_file/vendor_d/neuralwa/nw2v50/automate.txt

Text File | 1993-08-23 | 21KB | 445 lines

                   Documentation for the Automate Program


      Automate is a NeuralWorks User-Control program used to train or
      test networks in an off-line batch-processing mode.  It permits
      multiple networks to be processed (trained or tested) overnight
      without your direct intervention.  Automate requires version 5.0
      of NeuralWorks Professional II/PLUS or NeuralWorks Explorer
      software.

      Before running Automate you'll need to create it's data file
      'automate.dat'.  This is an ASCII file that contains your tasks
      for Automate to perform, and can be created using any text editor.


   Task summary:

      Automate understands the tasks (commands) listed below.  Shown is
      an example of each task, with square brackets denoting optional
      entries.

      open mynet[.nnd]            ! open an existing network
      savenet [newfile[.nnd]]     ! save the net (or save as)
      learn 10000                 ! train a network
      test                        ! test one pass/all
      recall                      ! recall one pass/all
      savebest 10000 500 10       ! savebest (train, test, auto-save)
      savebest 10000 500 10 .95 2 ! savebest with auto-pruning
      randomseed 257              ! reset the random number generator
      initialize                  ! initialize the network
      disable all                 ! disable each input PE and test
      disable 2 3 4 5 6           ! disable specific input PE(s) & test
      set learnfile train[.nna]   ! change the training filename
      set recallfile test[.nna]   ! change the recall/test filename
      set testfile test[.nna]     ! change the recall/test filename
      set disable                 ! disable specific input PEs (no test)
      holdout 1 3 .2              ! PNN & GRNN tests varying Sigma Scale


   Program execution:

      The following is an outline of steps needing performed during an
      Automate session:

      (1) Create and save a network using the Explorer or Professional
      II/PLUS programs.  Make sure your data files exist along with your
      network within the NeuralWorks directory.  You may want to partly
      train or test the new network to ensure it functions properly.

      (2) Create or modify automate.dat using any text editor.  The file
      should list the networks and tasks you wish to have performed.
      Make sure this file is saved in ASCII text mode, without word-
      processor formatting characters.

      (3) Start Automate running, perhaps waiting until later in the
      evening.  The NeuralWorks Professional II/PLUS and NeuralWorks
      Explorer command line to start a User-Control program varies
      depending upon the machine you are using:

          nw2  -xuautomate    ! ProII/PLUS on Unix and 286 IBM-PC
          nwe  -xuautomate    ! Explorer on Unix and IBM-PC
          nw2x -xuautomate    ! ProII/PLUS on 386 and 486 IBM-PC
          nw2s -xuautomate    ! ProII/PLUS on Solaris

      Automate is launched on a Macintosh by double-clicking on the
      automate application icon or name.

      See your NeuralWorks System Guide for more details on creating,
      compiling, or running User-Control programs.

      (4) After Automate execution, automate.dat contains 'Successful'
      or 'FAILURE' comments at the start of each task it processed.
      Checking automate.dat and nworks.err will indicate the cause of
      any unexpected errors.


   Usage Notes:

      Tasks can be abbreviated to their first 4 letters.  Savenet and
      Savebest can be abbreviated to 5 letters.

      Before processing any tasks, Automate checks your automate.dat
      file for syntax errors.  You are informed of any errors found and
      processing aborts.  Automate.dat marks the lines in error with
      'FAILURE' and a brief message about the nature of the error.
      Processing begins only if no syntax errors are found.

      Automate execution can be interrupted by pressing the Escape key.
      This halts the current task and (usually) asks whether processing
      should continue with the next task or be halted completely.
      Because the interrupted task is not completed, use this capability
      with caution.

      The Learn task automatically saves the network, in binary format,
      upon completion of training.  The currently opened network is
      overwritten.  You may want to make a backup copy of the original
      network before running Automate.

      Savebest also overwrites your current network and saves it in
      binary format.  A savebest-log file (*.sbl) is automatically
      written while this task is running.

      Savebest parameters in Automate are: Learn Count, Test Interval, #
      Retries, Pruning Tolerance and Max # of PEs to Prune.  The two
      pruning parameters need entered only if you wish to employ hidden
      layer PE pruning.  See the NeuralWorks Reference Manual for more
      information.  These parameters are space or comma delimited.

      Set Learnfile, Set Recallfile, and Set Testfile tasks change the
      name of the Learn or Recall/Test file within the I/O Parameters
      dialog.  This enables data processing from files other than those
      originally saved in the network.  These tasks apply only to the
      currently opened network, and if you save the network these
      'alternate' filenames are saved in the I/O Parameters dialog.
      These commands should probably not be used on networks that employ
      a UserIO program during training and testing.

      The Open task clears previously entered Set commands, so any Set
      tasks must follow and not precede an Open task.

      Automate's source code file automate.c is provided should you wish
      to review or modify the program.  If you make changes that are
      generally useful to others, please consider sending them to
      NeuralWare.  We will inspect and test the modifications, and
      possibly incorporate them into future versions.



      The remainder of this document discusses two special tasks,
      'Disable' and 'Holdout'.  These are examples of the kind of useful
      tasks than can be added into the program.


   The Disable task:

      Automate includes a technique named 'Disable', which is a method
      of analyzing a trained network to determine the importance of each
      input variable.  It has the same purpose as the NeuralWorks
      Run/Explain command, but with a whole different approach.  The
      Disable task will disable one or more INPUT LAYER processing
      elements, run a Test command, and record the R Correlation
      measurement (from one or more Confusion Matrixes) into a file
      named 'disable.log'.  This log file is appended to if it exists.

      Networks must be trained and specially setup for use with Disable,
      as described later.

      Automate.dat for a Disable run might look like this:

          open mynet
          disable 2 3    ! disable first 2 input PEs and test
          disable 3 4 5  ! disable 2nd, 3rd, and 4th PE and test
          disable all    ! disable each input PE one at a time,
                           testing at each step.  Only 1 PE at a
                           time is disabled using this task.

      Tasks 'disable every' and 'disable each' are synonyms for the
      'disable all' task.

      Note that Disable does not save the network; it only opens it,
      disables the specified input PEs, and runs a Test command to
      determine the drop in output performance when this PE (or group of
      PEs) is not providing input.  The specified PEs are marked as
      disabled and their Output values are set to 0.0, effectively
      removing this input node from the upcoming Test command.


   Disable results:

      Below is the disable.log results file produced from the above
      automate.dat file:

          !
          ! network:  audisabl.nnd
          ! sum R   ConfM1  ConfM2  ConfM3  %impact ---disabled PEs---
           2.7676  0.9963  0.8605  0.9106   0.000  (baseline)
           2.3815  0.9930  0.5518  0.8365  13.951  2 3
           0.0739  0.6572  0.1017 -0.6850  97.328  3 4 5
           2.7528  0.9887  0.8688  0.8953   0.531  2
           2.3342  0.9846  0.4992  0.8503  15.657  3
           2.0825  0.9073  0.5264  0.6487  24.753  4
           0.8115  0.9253  0.4447 -0.5585  70.676  5

      This result is from a partially trained Back-propagation network
      using iris_tra.nna and iris_tes.nna.  The network has 4 input PEs
      (PEs 2, 3, 4, and 5) and 3 output PEs.

      The first column of figures is the sum of columns 2 through 4.

      Columns 2 through 4, labeled ConfM1, ConfM2 and ConfM3, show the R
      correlation coefficient as displayed on a Confusion Matrix
      instrument within the network.  There is one Confusion Matrix
      graph per output PE.

      Column 5, labeled '%impact' shows the drop in network output
      occurring when the input PEs (shown in column 6) were disabled.

      The first numeric line above is a 'baseline' test, ran before
      disabling any input PEs. The baseline 'Sum R' is used by all
      subsequent tests to calculate each test's %impact.  If the %impact
      figure is negative, it indicates that the network results improved
      by disable that PE (or group of PEs).  The calculation for %impact
      is:

          %impact = 100.0 - (newTestSumR / BaselineSumR * 100.0);

      For example, disabling only input PE 2 (the first input PE) shows
      a much smaller drop (0.531%) in the network's predictive or
      classifying capability, relative to when PE 5 was disabled
      (70.676%).  You might infer from this that PE 5's inputs are
      substantially more important to the network, in a global sense,
      than are PE 2's inputs.  Closer inspection of the entry for PE 5
      also shows a much smaller affect on the first output class than on
      the second and third outputs.

      If Disable results (or results from the Run/Explain command)
      indicate that one or more inputs can be deleted with minimal
      impact the network's performance, you should re-train a new
      network without these inputs.  This will verify whether the inputs
      are indeed unnecessary.

      For both Disable and the Run/Explain command, you might wish to
      run separate tests using (1) the training file, (2) a test data
      file, and/or (3) a combined file of training and test data.  The
      combined file is likely to show a more global picture of things
      than when testing only a subset of your data.


   Network setup for Disable:

      The Disable task relies upon Confusion Matrix instruments to
      calculate the R correlation value, and expects this information to
      be found in a temporary file named 'disable.nnp'.  This is
      accomplished within the network (after it is trained) by doing the
      following:

      (1) For each output PE you are interested in monitoring, create a
      Confusion Matrix instrument for that PE. Using the Graph/EasyProbe
      icon, select "Output Value", "Selected PE", "Transform across
      Epoch", "ConfM", and okay the dialog.  Apply the graph to the
      output PE of interest, and repeat the process for all output PEs
      you are interested in monitoring.

      (2) Edit each Confusion Matrix graph.  Select the "logging active"
      button, and enter "disable" into the filename field.  The "append"
      selection should be used, and not "write".  This will cause all
      graphs to record their R values into disable.nnp.

      (3) VERY IMPORTANT: Unless you want very misleading results, make
      sure that multiple Confusion Matrixes are edited in the 'proper
      order'.  Graphs plot in the order they have been accessed in, and
      so the logged values within disable.nnp will be in this same
      order.  After setting up the graphs and editing their contents,
      perform a Graph/Edit (or double-click) on each graph in sequence,
      starting with the graph for the first output layer PE, then the
      second graph, etc.  Simply bring up the Graph/Edit dialog and okay
      it.  If in doubt, press the Escape key and the screen will
      repaint.  Watch the order that the graphs appear; they should be
      in a left-to-right fashion.

      (4) Do step 3, or the results will be very misleading and you
      probably won't realize anything is wrong.


   Specifying PE(s) to disable:

      Only Input layer PEs can be disabled using the Disable task.

      Up to 50 PEs can be disabled at one time, separated by spaces or
      commas.

      The first PE in an Input layer is labeled "2" on the screen, and
      is known as PE 2 (the "Bias" PE is PE 1, and it is not in the
      Input layer).  Entering the number 2 into the PE-list therefore
      disables PE 2, the first Input layer PE. Likewise, entering 100
      disables PE 100, the 99th Input.


   The Set Disable task:

      'Set Disable' is a special task with rather esoteric use.  Suppose
      that after running Disable as shown above, you believe inputs 1
      and 2 (PEs 2 and 3) are not contributing much to the output of the
      network.  To verify this, these PEs need to be disabled, and the
      network needs to be initialized and retrained.  You can either
      disable the PEs manually within the Explorer or Professional
      II/PLUS using PE/Edit, or you can use 'Set Disable'.  This setting
      tells Automate to disable the PEs, without running any tests.  It
      is assumed you will then Initialize and Train the network, or
      'Savenet' the network.  Automate.dat for this task looks like
      this:

          open mynet
          set disable
          disable 2       ! could be one command 'disable 2 3'
          disable 3
          savenet mynet2


   The Holdout task:

      This task exists only for use with Probabilistic Neural Network
      (PNN) and General Regression Neural Network (GRNN) paradigms, and
      will not work for any other network type.  Due to these paradigms
      using 1 hidden layer PE for every training data case, and because
      they are often used when few data cases are available, they lend
      themselves to a testing technique known as the 'hold-out' method.

      During 'hold-out', the network is trained with all data cases
      except one which is used for test purposes.  The network results
      are recorded, and the network is trained and tested holding out a
      different data case.  The process repeats for each data case.

      Although this method may be valid for any paradigm when few data
      cases are available, the topology of PNN and GRNN networks makes
      this method very easy to automate, and does not even require
      re-training the network.  After initially training the network,
      only one pass through the data set is required while disabling the
      appropriate 'pattern layer' PE for each data case.

      Automate's Holdout task differs a bit from the above description.
      It is not primarily intended for use in situations with few data
      cases.  Instead it is concerned with optimizing one of the L/R
      Schedule coefficients, the Sigma Scale setting.  Holdout requires
      three parameters (as explained later) to explore various Sigma
      Scale settings, and produces results that indicate the
      near-optimal value for this coefficient.

      Your network should be trained before using Holdout, and it must
      not be using a UserIO program for training.

      Holdout works by running Test commands using single data cases
      extracted from your training data file.  Your network may have a
      separate Test data set, but the Test data is not used by Holdout.
      Internally, Automate looks only at your training data.


   Holdout parameters:

      The Holdout task parameters explore various Sigma Scale settings.
      You supply the start, stop, and step size for this setting, and
      Automate performs one Holdout task for each setting.  The three
      arguments are entered space or comma delimited.

      If the first parameter (the starting Sigma) is entered as a
      negative value, then the start and stop Sigma becomes an offset
      from the network's current Sigma Scale setting.  A positive
      starting Sigma is considered an absolute range; negative is an
      offset, relative to the current Sigma Scale setting.

      For example, a task input line of 'holdout 1.0 3.1 0.5' will test
      six different Sigma Scale coefficients {1.0, 1.5, 2.0, 2.5, 3.0,
      3.1}, regardless of the Sigma Scale setting currently set within
      the network.  A task input line of 'holdout -1.0 3.1 0.5' tests
      ten different Sigma Scale coefficients.  If the network's current
      setting is 4.0, then the tested settings are {3.0, 3.5, 4.0, 4.5,
      5.0, 5.5, 6.0, 6.5, 7.0, 7.1}.  The minimum Sigma Scale tested is
      0.0001, which is set by Automate if your range tries to set a
      smaller value (Sigma Scale cannot be less than or equal to zero).


   Holdout task results:

      The Holdout task causes Automate to perform a series of hold-out
      tests rather quickly, and outputs an overall measure of the
      network's performance into an output file named 'holdout.log'.
      This file is appended to if it already exists.

      For PNN networks, this measure is the 'average classification
      rate', the same measure as shown by our Classification Rate
      instrument.  This instrument counts the percent-correct for each
      output class, and reports the average of these scores.  A value of
      0.0000 is terrible, and 1.0000 is a perfect score on all classes.
      Further documentation for this instrument can be found under the
      Graph Palette in the NeuralWorks Reference Manual.

      The measurement employed for GRNN networks is the root-mean-square
      (RMS) of all errors.  A lower RMS error is better than a higher
      one.

      Holdout.log results indicate whether you should continue to
      increase or decrease the network's Sigma Scale setting, or whether
      you should run additional experiments with smaller step sizes
      (increments) to explore a more localized space.

      A sample automate.dat file for Holdout is shown below:

          open my_pnn
          holdout .5 1.5 .2

      The resulting holdout.log file is shown below.  The results
      indicate that the optimal Sigma Scale is somewhere between 0.5 and
      0.9, and so additional Holdout tasks should be run in the range of
      0.5 to 0.9, with a smaller step size.

          Holdout testing of network my_pnn
          Sigma Scale varies from 0.5000 to 1.5000, stepped by 0.2000
             avg classification rate = 0.7533, sigma scale = 0.5000
             avg classification rate = 0.7936, sigma scale = 0.7000
             avg classification rate = 0.7655, sigma scale = 0.9000
             avg classification rate = 0.7412, sigma scale = 1.1000
             avg classification rate = 0.7412, sigma scale = 1.3000
             avg classification rate = 0.7384, sigma scale = 1.5000

      At completion, Holdout places the 'best' Sigma Scale value back
      into the L/R Schedule.  Holdout does not alter your saved network,
      unless automate.dat includes a Savenet task.  Saving the network
      then writes the network with only the Sigma Scale coefficient
      being different than the original network.

      And finally, consider the following automate.dat file:

          open my_pnn
          holdout .5 1.5 .2
          holdout -.4 .4 .05
          holdout -.1 .1 .005
          savenet my_pnn2

      The first Holdout task causes Automate to test between 0.5 and 1.5
      as shown earlier, and a new Sigma Scale coefficient of 0.7 was
      placed into the L/R Schedule (because 0.7 generated the highest
      average classification rate).  The second Holdout task starts with
      0.7, offsets this (because of -.4) and tests the range 0.3 to 1.1
      with a smaller step size, again placing it's best Sigma Scale into
      the L/R Schedule.  The third Holdout task potentially refines it
      even further.  The Savenet task saves the network as 'my_pnn2.nnd'
      with a fairly optimal Sigma Scale setting.

      If the Holdout task fails (i.e. 'FAILURE' is written into file
      automate.dat), check nworks.err and also the holdout.log file to
      determine the cause.

      Automate will run faster during Holdout tasks if the test data set
      is loaded into Ram, and also if all on-screen graphs are turned
      off during Recall mode.

      For more information on the 'hold-out' method and the Sigma Scale
      coefficient, see the PNN and GRNN chapters in your Neural
      Computing Guide, and the on-line help screen for the InstaNet menu
      for these paradigms.


   End of automate.txt